Introduction to Machine Learning

Unit 10: Naive Bayes (Continued)

1. Introduction

This second lecture on Naive Bayes deepens the practical side: we implement NB's smoothing step-by-step as an in-class activity, run a Python end-to-end demo with scikit-learn, compare the three major NB variants (Multinomial, Bernoulli, Gaussian), and study how text gets vectorized (Bag-of-Words, TF-IDF) for NB-based spam detection. We close with a bias-variance-style decomposition of NB's error and a three-dataset benchmark comparing NB against kNN and Decision Trees.

Learning Objectives

2. Theory

2.1 In-Class Activity — Laplace-Smooth All Play-Golf Features

Recall the Play-Golf dataset with 5 "No" training rows. We already smoothed Outlook for "No" (|V|Outlook = 3). Complete the remaining three features for the "No" class with α = 1:

Temperature
Humidity
Windy
✅ Answers

Raw counts (Temp ∣ No): Hot=2, Mild=2, Cool=1. Total=5. |V|Temp = 3.

Compute smoothed P(Temp=Hot∣No), P(Mild∣No), P(Cool∣No).

Raw counts (Humidity ∣ No): High=4, Normal=1. Total=5. |V|Humidity = 2.

Compute smoothed P(High∣No), P(Normal∣No).

Raw counts (Windy ∣ No): False=2, True=3. Total=5. |V|Windy = 2.

Compute smoothed P(False∣No), P(True∣No).

2.2 Three Flavors of Naive Bayes

VariantLikelihood ModelTypical FeaturesUse Case
GaussianNB \(P(x_i \mid y) = \mathcal N(\mu_{y,i}, \sigma^2_{y,i})\) Continuous numeric (cm, kg, °C) Iris / Wine / Breast-Wisconsin
MultinomialNB \(P(x_i \mid y)\) from normalized counts Word counts, integer frequencies Text (spam, sentiment, newsgroups)
BernoulliNB \(P(b_i \mid y) \in (0,1)\), binarized features Binary / presence-absence Short texts, binary user features

⚠ GaussianNB Numerics

GaussianNB fits a per-class per-feature normal distribution. For numerical stability, scikit-learn adds a tiny epsilon \(\epsilon=10^{-9}\) to every variance so no variance is ever exactly zero. Always scale/standardize numeric features if you want all features to contribute comparable Gaussian log-likelihood magnitudes.

2.3 From Raw Text to NB — Vectorization

NB cannot operate on strings. We first convert each document into a fixed-length numeric vector.

E-mail text feature extraction pipeline An e-mail string is tokenized and then represented using Bag-of-Words, Binary Bernoulli, or TF-IDF features. Raw e-mail string Step A: TOKENIZE (lowercase, split) "WIN!! Free Money NOW" → "win free money now" 1 Bag-of-Words (CountVec) count(word) per doc d = |V| integer cols 2 Binary (Bernoulli) 1 (word present) per doc d = |V| binary cols 3 TF-IDF TF × IDF score per word reweights common words DOWN

2.4 TF-IDF Formalized

Term Frequency × Inverse Document Frequency weights down words that appear everywhere (the, a, of) and boosts words that are rare and hence discriminative.

\[ \text{tf-idf}(t, d, D) = \underbrace{f_{t,d}}_{\text{TF}} \;\times\; \underbrace{\log\frac{|D|}{|\{d' \in D \mid t \in d'\}|}}_{\text{IDF}} \]

2.5 Python scikit-learn Spam Pipeline (Conceptual)

from sklearn.pipeline import Pipeline from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import roc_auc_score pipe = Pipeline([ ('vect', CountVectorizer(stop_words='english', min_df=5, ngram_range=(1,2))), ('tfidf', TfidfTransformer()), ('clf', MultinomialNB(alpha=0.5)) ]) pipe.fit(X_train, y_train) y_proba = pipe.predict_proba(X_val)[:, 1] print(f"Validation AUC: {roc_auc_score(y_val, y_proba):.4f}")

2.6 Benchmark — NB vs. kNN vs. Decision Trees

DatasetMetrickNN (k=5)NB (Gauss/Multi)Tree (depth 5)
Iris (num, 4f) Accuracy0.9670.960 (Gaussian)0.953
SMS Spam (text) AUC0.820.985 (MNB)0.90
Adult (mixed, 14f) AUC0.830.86 (MNB)0.88
Training time (relative)  10×1×3×
🔍 Benchmark Observations (click to expand)
  • NB wins big on text (SMS spam) — the conditional-independence assumption is surprisingly effective when features are words.
  • All three methods are competitive on clean, low-dimensional numeric data (Iris).
  • Decision Trees pull ahead on mixed tabular (Adult) because they model non-linear feature interactions and splits — something NB cannot do.
  • NB is consistently the fastest trainer by an order of magnitude — strong as a baseline first model.

2.7 When NB Works (and When It Fails)

👍 NB Shines When👎 NB Struggles When
Small training sets (low variance)Strongly correlated features exist
Text / high-dimensional sparse inputsYou need calibrated probabilities (use Platt scaling)
Streaming / incremental updates requiredFeature interactions drive the prediction
A low-compute baseline is neededNum features is tiny & signal is all-interaction

3. Interactive Examples

Example 1: GaussianNB on Iris

Two-class (Setosa vs. Virginica) slice of Iris. Fitted per-class Gaussian parameters (Petal-Length cm): Setosa: \(\mu=1.46, \sigma^2=0.03\); Virginica: \(\mu=5.55, \sigma^2=0.30\).

(a) A new flower has Petal-Length = 3.0 cm. Which class does GaussianNB favor?

Compute log-likelihood ratio using \(\log \mathcal N = -\frac{(x-\mu)^2}{2\sigma^2} - \log\sigma\). Ratio favors Setosa over Virginica by ~2.5 nats → predict Setosa. (3 cm is 5σ away from Virginica's mean, but only ~8σ from Setosa — Virginica's larger variance softens the blow but not enough!)

(b) Why is \(\sigma^2_{Virginica}=0.30\) so much larger than \(\sigma^2_{Setosa}=0.03\)?

Virginica petal lengths are genuinely more spread out in nature than Setosa's (which are tightly clustered). GaussianNB learns different per-class per-feature variances and uses them correctly.

Example 2: TF-IDF Intuition

A 10,000-document email corpus. "the" appears in 9,900 docs; "viagra" appears in 100 docs.

Compute IDF("the") and IDF("viagra").

IDF("the") = log(10000 / 9900) ≈ 0.01   (near-zero weight)
IDF("viagra") = log(10000 / 100) ≈ 4.605   (~460× more discriminative weight!)
TF-IDF therefore essentially drops stop-words from the classification automatically, even without a stop-word list.

Example 3: Benchmark Choice

A startup ships a spam filter on a Raspberry Pi (very low CPU) and must retrain daily on 100K new labeled emails. Accuracy is "good enough" at any score ≥ 0.95 AUC; training-time budget: 30 seconds.

Choose the best model from {kNN, GaussianNB, MultinomialNB, DecisionTree} and justify in 1 sentence.

MultinomialNB with TF-IDF: text input → multinomial is correct; MNB trains 10× faster than kNN and hits ≥ 0.98 AUC on SMS spam in the benchmark — comfortably above 0.95 within the time budget.

4. Numerical Solutions

Problem 1: GaussianNB on 2-Class 2-Feature Toy Data

Class A (n=3): samples \((1,2), (2,3), (3,4)\)  ·  Class B (n=3): \((6,7), (7,8), (8,9)\).

📘 Step-by-Step Solution — Classify (4, 5)

Step 1: Class priors equal (3/6 each = 0.5).

Step 2: Fit Gaussians.

  • μA = (2, 3), σ²A = (2/3, 2/3) ≈ (0.667, 0.667)
  • μB = (7, 8), σ²B = (2/3, 2/3) ≈ (0.667, 0.667)

Step 3: Evaluate log-likelihood at (4,5):

  • log P(A) contrib = −(4−2)²/(2·0.667) −(5−3)²/(2·0.667) ≈ −6.00
  • log P(B) contrib = −(4−7)²/(2·0.667) −(5−8)²/(2·0.667) ≈ −13.50

Step 4: argmax → Class A (by ~7.5 nats). Equal priors → pure likelihood fight, and (4,5) is closer to A's center.

Problem 2: Laplace-Smoothing Parameter Sweep

Rare-word case: vocabulary size |V| = 10,000. In Class Y=+, a rare word "antidisestablishmentarianism" has count(word, +) = 0 and count(+) = 1000.

📘 Step-by-Step Effect of α

\(P_s(w\mid +,\alpha) = \frac{0+\alpha}{1000 + 10000\alpha}\). Evaluated at different α:

  • α = 0 → 0 (broken — zero frequency problem)
  • α = 1 → 1/11000 ≈ 9.1 × 10⁻⁵
  • α = 0.1 → 0.1/(1000+1000) = 5 × 10⁻⁵
  • α = 10 → 10/(1000+100000) ≈ 9.9 × 10⁻⁵

Small α = trusts the data more (closer to MLE); large α = smooths toward uniform 1/|V|. Best α is tuned on a validation set!

Problem 3: TF-IDF Ranking

|D| = 100,000. Document d1: "cat cat dog" (3 tokens); d2: "the the the cat" (3 tokens).
Document frequencies: DF("the") = 80,000; DF("cat") = 5,000; DF("dog") = 4,000.

📘 Step-by-Step — Which word dominates d1 vs d2?

Step 1: IDFs: the→log(100k/80k)=0.223; cat→log(20)=2.996; dog→log(25)=3.219

Step 2: d1 tf-idf scores: cat = (2/3)·2.996≈1.997; dog=(1/3)·3.219≈1.073

    → "cat" dominates (high TF × high IDF).

Step 3: d2 tf-idf: the=(3/3)·0.223≈0.223; cat=(1/3)·2.996≈0.999

    → "cat" dominates again despite lower TF because "the" is crushed by its tiny IDF.

Conclusion: TF-IDF robustly recovers "content words" over stop-words, even when stop-words are the majority of tokens.

5. Try It Yourself

Problem 1 — GaussianNB Classification

Classes: A(μ=0, σ²=1), B(μ=4, σ²=4), equal priors. Classify x = 1.5.

  1. Compute log-likelihood for A and for B.
  2. Which class wins? By how many nats?
  1. log ℒ(A) = −(1.5)²/2 − log(1) ≈ −1.125; log ℒ(B) = −(1.5−4)²/(2·4) − ½·log(4) ≈ −0.781 − 0.693 ≈ −1.474.
  2. Class A wins by about 0.35 nats (despite B having a flatter wider Gaussian, x=1.5 is much closer to 0 than to 4).
Problem 2 — BernoulliNB Play Golf

We convert the 4-category Outlook feature into 3 Bernoulli dummy features (IsSunny, IsOvercast, IsRain). P(IsOvercast∣No, α=1, |V|=2 per feature) = ?

Hint: We're now working per dummy, so vocabulary size is 2 (true/false). The No class has 5 training rows.

count(IsOvercast=T, No) = 0, count(No)=5. Pₛ = (0+1)/(5+2) = 1/7 ≈ 0.143.
(This is the same "rare event with smoothing" situation — BernoulliNB dummies just make each binary feature explicit.)
Problem 3 — NB Incremental Throughput

Batch 1: 1,000,000 docs (700 K spam, 300 K ham). Count("sale"∣spam) = 200 K; Count("sale"∣ham) = 6 K.
Batch 2: 100,000 new docs arrive. Count("sale"∣spam) = 18 K; Count("sale"∣ham) = 500.

(i) What are the merged counts? (ii) What are the merged P("sale"∣spam) and P("sale"∣ham) without smoothing?

Merged spam docs: 700K+? — need to solve Batch 2 spam/ham split!

Assume Batch 2 class distribution is 50K spam / 50K ham for the problem:

  • Merged spam = 700K+50K = 750K. Merged ("sale"|spam) = 200K+18K = 218K → P=218K/750K ≈ 0.291.
  • Merged ham = 300K+50K = 350K. Merged ("sale"|ham) = 6K+500 = 6,500 → P=6.5K/350K ≈ 0.0186.

LR("sale") ≈ 0.291/0.0186 ≈ 15.6× strong spam signal.

6. Interactive Quiz

Answer all 5 MCQs. Click on an option to get instant feedback.

Your score: 0 / 5

7. Key Takeaways

  1. Laplace smoothing applies to every feature. Use the vocabulary size \(|V_i|\) of that specific feature in the denominator. Don't reuse one feature's |V| for another.
  2. 3 NB variants for 3 data types: GaussianNB = continuous features; MultinomialNB = integer/count (bag-of-words text); BernoulliNB = binary presence/absence.
  3. Text → fixed vectors via CountVectorizer or TF-IDF. IDF crushes near-universal words (the, of, and) automatically, letting rare discriminative words dominate.
  4. NB is speed king for text — 10× faster than kNN, 3× faster than shallow trees, with best-in-class AUC on text. Use NB as your first baseline before trying expensive models.
  5. Incremental merging is exact. Add frequency tables element-by-element for every mini-batch. Recomputing from scratch is wasteful and unnecessary!
  6. Bias-variance intuition: NB has high bias (strong independence assumptions) but extremely low variance — it wins in small-data / high-d regimes where low-variance methods dominate.

8. Common Pitfalls

  1. Using GaussianNB on bag-of-words counts. BOW integers are not normally-distributed — use MultinomialNB instead. The mismatch usually costs 5–10 % AUC.
  2. Applying raw CountVectorizer without min_df / stop-words / pruning. 10⁵+ vocabulary blows up memory; hapax legomena (words seen once) hurt generalization.
  3. Sharing the same α across all NB variants blindly. α=1 (Laplace) is a default; for MultinomialNB on text, tune α ∈ {0.1, 0.5, 1, 2} on a validation set to squeeze out 1–2 % AUC.
  4. Interpreting NB's predicted probabilities as well-calibrated. The independence assumption distorts magnitudes. Use Platt scaling / isotonic regression via CalibratedClassifierCV if calibrated probabilities matter.
  5. TF-IDF on already-normalized likelihoods. Apply TF-IDF to the raw count matrix, then feed the reweighted matrix to MultinomialNB — don't try to apply it after NB training (too late).
  6. Benchmarking a single train-test split only. NB is stable but always use 5-fold CV with fixed random seed when comparing models — a lucky split can easily lie by 3 %.

9. Resources